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ABSTPACT 

. ' , , A user's guide for the McGill Evaluation system of 

•instructional evaluation and a bibliography are presented. The guide 
covers the initial planning of an evaluation, data collection,, and 
wafcing changes in a course or prograu. The procedure used in the 
svstes coT^tains the following sequential steps: describing the 

hr£S!?„^£J^^^r^T.^°''' decidinq the aspects of the instruction to 
be evaluated, determining what inforaation will be collected,, setting 
criteria for change or decision, collecting, analyzing, and 
Interpreting inferaation, and aalcing changes, steps in collecting 
inforaaticn are to deteraine sources of inforaation and techniques 
-or data collection, and to select or develop instruaents- Research 
instrnaents and technique^ that aay be used are described. Selecting 
criteriji for teacher and/or course improvement, prograo evaluation,^ 
or personnel evaluation are considered. Brief cooaents are presented 
regarding aethods for analyzing inforsation--<pr several ^ypes of 
instruaents or technigues. The tiae and resources available to 
actually iapleaent chanlges and to plan in detail what will be done 
^ith the evaluation results arc reyiewed. (SW) 
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BrUharfe> J.K. Effective group discussion (2nd ed.). Dubuque, 
\ IoWa:.WrD. C. Brown Company, 1971 ,,95- 149. 
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\ . ■■ • 



RESOURCE FILE IE ^ 
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? • • ■ 

Ci&^hen, A. - Objectives for college cpuVses.. Beverley Hill-s. 
Calif.: Glencoe Press, 1970, 9-6M, 73-87. 
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D.C.: Aaieri.an Association for Higher Educatioi, ^gys!"^'^""' 

^^"^'*ri^^;»; . as godfather? The impact of student 
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*■ -."".■* 

Feldman, K.A. Course characteristics and colleKe students' 
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Research in Higher Education . 1978, 9, 199-242. 
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in Education . 1975, .3, 210-2U0. ' 
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Donald, 4.G. , The evaluation of learning. Learning and 
' Development . 19-76, 8, (2).^ • . ' .bH£niiIl5 Blil 

Roid.^G.H. Towards a system of course evaluation. Learning and 
Development . 1971, 2, (2). ■ ^ 4^ 
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in!^^ problems without the use of standardized 

instrvuments Paper presented at the Annual Meeting of the 
American Educational Research Association. 197^ 
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Burnett,- J.D. Evaluating teachers. Paper presented at the 
annual meeting of the Canadian Society for the Study of 
Educat-ion, Edmonton, Alberta, I975. u uy 

Gronlund, N. Checklist for. evaluating a criter ion-referenced 
test. Preparing criter ion- referenced tests for classroom 
instruction : Wevi York: KacMillan, 1$"r3 , $1-5 2: — ^ 

A "odel for a course syllabus. Teaching Skills Series, No. 10 
Teaching Seneca College; Centre for LearnlKg iFd 
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Bogdan, R., & Taylor, S. Introduction to qualitative rf^search 
nethods. New York: John Wiley & S6ni7 1?t5, 8^-^17 "n^TiTrr 

Sherman, T.M., & Winstead, J.C. A formative approach to student 
15 3?!39"'*°^ instruction. Educational Technoloj^y . 1975, 
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^^''^^d.,?;!!** S^^'r^^^'u J:-'^" * Yelon,^S.L. Learning system 
desi^. New York: McGrawHill, 130-19^7"^ — 

^^^^^1968*"6 i^g^'""^"8 hierarchies, Educational Psychologist . 

^^*» ^- , ^g^'I.^QQl fourid f.tions of education al research. 
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McCallon, E.L., & McCray, E^- Planning and conducting interviews. 
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York: The Psychological CorpoFation, 1^5. 

Sax, G. Obtaining InforBkation from respondents: I-the interview 

^pia^^nh^'^^^r^''?"^^'*!- foundations of educational 

research. Englewood CTmrTHTJ,: Prentice"- H all , 1^68, 
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RESOiJRCE FILE 3^ 
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Instructional development service project: Develooment of a 
>. £ate£ory observation sj^stem. McGill Unlv ersityt^lrg . — ^ ~ 

Martin, J, ' The development and- use of claffsroo^ observation 
inatri^ents. Canadian Jour^ 1977? I! !! 3 -5 4? 
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. |evelopment. Washington, D.C.rThi gfunciT^fo r Ihe 

Advancement of Saall Colleges, 1975, 52-86. 
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University of Illinois 
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Illinois Course Evaluation Questionnaire. '■^'^ 

Instructor self evaluation for (Self-report form to acc^paW 
b.l.j?. form, Educational Testing Service). 

^^''^^Vgi^^/^' Personalized Course Analysis Procedure, 

Wood, P. The description ,.and evaluation of a collefff* 

anSu^?'"""''?/'^"''?: '"^^^^S system. ^ Pao.r presented aJ the 
annual meeting of the American Educational - Research 
Association, New York, April r977. «"«a^it?nai research 

''°''^"?n»trM;Mo„=f''^^i""'^"8 Itiatruotional effectiveness with the 
SoSJheri ^questionnaire. Corbondale? 

undated University, Counselling A Testing Centre. 
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ARTICLES ON HOW TO W^ITE ITEMS FOR QUESTIONNAIRES 
AND HOW TO CONSTRUCT THE FINAL QUESTIONNAIRE 

Tlener^on, M., Morris, L, & Fitz-gibbon, C. How lo measure 
attitudes . Beverley Hills: Sage, 1978. ^ . ~ 

McCallon, E. , & McCray, E. Designing and using questionnaires. 
Austin, Texas: Learning ConceptsT TWS. 

Sanders, J., & Cunningham,^ D. Formative evaluation: selecting 
• ^ techniques gnd procedures. In S". Borich, (Ed.), Evaluating 
educational prograpis and products . Educational technology 
Publication, 1974. 

Sax, G. Empirical foundations of education al research. 
Englewood Cliffs, N.J.: Prenticeliall, l968,*'2f1Ti"-23T: ' 

ZimbarMo, P., & Ebbeson, E. Influencing attitudes and changing 
behavior . Reading, Mass.: Addispn-Wesley , 1$68,n?3^n7F. — 
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Cranton, P. A. Construc ting testa . Centre for Learning and 
Development, 1977. gee Teaching and Learning Modules . No. 

Green, J. A. " eacher - made testSf- New York: Harper & Row, 1975, 
85-1 35. 
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* 

McCallon, E.L.,. & McCray, E. Planning and, conducting interviews. 
Austin, Texas: Learning Concepts, IWfSZ 

Morgan, H., & Cogger, J. The interviewer * s manual. The 
Psychological Corporation, T973. ~ ' 



Sax, G. Empirical foundations of educational research. 

Englewood Cliffs, N.J.: Prentipe^all , 1968, — ^ w 
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RESOURCE FILE 5C 
ANALYZING QUESTIONNAIRES 



Cameron, A. M, Multiple choice examinatio n and course evaluation 
"ser^ s gui^ McGill Cooputifhg Centr eT^AugustTgTTT''^ * 

Kulik,. I. A. Student reac'tions to ', instruction . Motnc ^o the 
Faculty #58. U'niversity of Michigan, Centre foFT^esesT^ch^on 
learning and Teaching, October 1976. 



RESOURCE FILE 5D 
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Teaching Effectiveness Coninjittee of the CAUT, Guide to ♦•he 
Kf.£^^"S^ dossier: its preparation and use, WJK WipoFFT 
GAUT, 1977. ~* 
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MCGILL EVALUATION SYSTEM ^ER'S GUIDE • 
• ' ■ * ' 

Desc ri ption 

• • ^» 
- The McGill Evaluation System i» .a practical step-by-step 

procedure for- university faculty who are interested in 

A 

instructionai evaluation, -^his guide will take you from the 
initial planning of an evaluation, ttirough data collection, to 
the making of .changes in a course or program. ; Supplemental 
materials and readings are contained in resource' files which 
accompany each step, and are referred to in the guide. You will 
also be directed to other services or facilities on the campus if 
necessary. It is recommended that you work with a consultant 
fro.fl the Centre fc^JJg^ching and Learning Services (CTLS) 
however, the eUent of his or her involvement will va.ry. 

Jhe procedure -used in this system contains six sequential 
steps.^ i 

.1. OesSbrifbing the purpose of the evaluation. 
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Deciding the aspects of the intruct'ion to be 
evaluated. ' 



3. . Determining what information will be collected. 

4. Setting criteria for change or decision. 

5. Collecting, analyzing and interpreting the 
Information. 

' 6. Making changes. * ^ 



If) 



Procedural and Policy Issu es 
During the first two years of the implementation of the 
McGlll Evaluation System, several issues arose regarding its use. 
Through discussion at CTLS, consultation with faculty members, 
and with other instructional development centres, conserves was 
reached on the following items: , •. . . 



1 . 



The professor or department who undertakes an evaluation 
is responsible for the planning and implementation of 
the procedures'; Centre for Teaching and Learning 
Services staff act ' as consultants rather than 
evaluators; that is, advice and resources are supplied 
whenever required, but the actual evaluation is done by 
the client. 

Participation in any evalua^:ion is completely voluntary; 
it is. recommended that any individual being evaluated 
also be Involved in the entire planninf? process. 
Ev.^l^j/^ion result-; are confidential, and are - the sole 
property of the professor conducting the -evaluation. 
When a" department-wide evaluation is being conducted, 
each individual professor is responsible for the 
dissemination and use of his or her results. However, 
an agreement may be reached within a department or 
between a professor anTd a chairman specifying the 
distribution of evaluation results. ' 
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M; Evaluations for personnel decisions based on the. 
Evaluations System must contain a statement from .1 
consultant at the Centre for Teaching and le.irnlnR 
Services attesting to the strengths and limitations of 
the procedures used. The Centre accents no 
responsibility for evaluations whigh do not conf-orm to 
thi s pol icy . 

Overview of the Evaluation System 



What is the 
purpose? : 



I!- 



What aspects 
of instruction? 



J Consultants and 
Resoui|tie Files 
— 7 1 



What information? 



■ ' " ■ 'J ' ' / 



What criteria 
will be used? 



/ ^ 
I f 

/ / 



Collecting the 
in formation 



Making 


dec isions 


an-d 


changes 
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For ^:\rh step you will be Riven the following 
in format ion : ' ^ • 

♦ 

1. The Task: Wh-^t you will do at this step, and how it 
relates to the evaluntion process as n whole. 
Hv)w to (\)mpl**te the . T-isk: Aot.ivities, r osoiir*<*^^>s > 
r t^Ven^ncor, ^ mattMinls tn ur;e, people to consult. 
Is thf» T.'^sk Complete? A' summary of wh.it you' should hav^ 
at the end of the step. 



STEP ONE ■ 

^ 

. WHAT IS THE PURPOSE OF THE EVALUATION? 
• •' ThG task 

Han.y details in tl^e evaluation procedure will vary depending 
on your purpose; it is important, therefore, to be as specific as 

v' 

possible, and to reach consensus if more than one professor is 
involved. 

When this step is complete, you should have answered the 
following questions: , * ' 

1. What is prompting the evaluation? 

2. Will the evaluation be primarily for improvement or for 
personnel decisions? % 

3. What types of changes are -^ou considering, or what 
decisions do you need to make? 

a. What will be evaluated (teaching skills, course content, 
course organization, program organisation)? 

It is rr^commonded that you record-your goals or^ purposes in order 
"^to reff»r bank to them at- later stages of the evaluation. 

How to Complete the Task 
Question 1: Whatsis prompting the^evaluat io n? 

r 

The answer to this question should be concrete and 
straightforward. ^" 
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students in the third ye'.ir of the program nre not 
adequately prepared. 



The failure rate among students is high. 

3. I am applying for promotion next year. 

« 

4. I want to see what improvements T can make in my 
teaching . 



Q^jg-g.yL'^l- = Will the evaluation be pri^marily for improvement 
- or- personnel decisions? 

Although evaluation for improvement and for personnel 
decision-making overlap - in many instanc'es, there will be some 
variation in procedure, e.g., personnel decision-making may 
require more comprehensi v^^ and more general information; 
evaluation f^ir improvement could be an in-depth examination of 
one aspect of a course. It is necessary, therefore, to specify 
the primary purpose of the evaluation before planninfj of the 
procedures' begins. Other "administrative" decisions such as 
changes in the content of a program can be viewed as having the 
overall purpose of improvement and thecefore will be categorized 
as "ev alxiation for improvement." 

Articles in Resource File 1A contain further thoughts on the 
improvement-administrative distinction. 

Baker, R.L. Curriculum evaluation. Review of Educational 
Research , 1959, 39(3) ~ 

Distinguishes between formative (improvement- 
oriented) and summative (decision-oriented) evaluation. 
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Cronbach, L.J. EvaluaticJn for course improvement. Te.icher's 
College Record . 1963^64, 67?-683 . ' 

Grasha, T. Pr inc iples and model s for assessing faculty 
performance : a monog raph ; Cincinnati : Faculty Research" 
Center, University of Cincinnati, 197?. (File 2C) 

This article presents different modjels .of 

evaluation with suggestions for implementation. 

Roid, C.H. Systems design for cours-e evaluation." Paper 
presented at ^ the annual m^etiTig of the American 
Educational Research Association, Chicago, April, 1972. 

Pages 1-8 provide a good general introduction" to 
evaluation. 

• Qugstion What types of changes ^re yo^ considering or what 

decisions do you need to make? 

The mast difficult aspect of this question is to focus or 
direct; i.e., to move away from generalities such as "We want to 
improve the program." 

The answer shvTuld be specific enough to allow you to .judge 
th^ f'-T.sibil ity of your planned changes in terms' of time and • 
resour.^e.; . Will you, for example, enroll in a '10-hour course to 

improve your teaching, • or will you spend 2 weeks during the 

f- 

summer reorg an i zi^ng your course? , 

t 

Example Answers « 

1. A committee will be established to rewrite the 
objectives and the course outline for the introductor*y 
course. 

2. Professor X will read a number of articles on teaching 
methods and enroll in a sel f- in str uctional course on 
teaching skills. 

3. A new format will be tried for the course (e.g., 
computer-assisted instruction, or student-led sem inar s)\! ' 
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M^ny educators have examined the process of change at the 
university leve'l — the factors that make a change acceptaljle, ?ind 
how to evaluate the effect of the change. ^You may find these 
analyses useful in judging the types of change^ that are feasible 
for your own situation. File IB contains: 

ft 

Corprew, J.C.,- & Davis,. K.J. An organizational development 
effort to improve instruction at ^ university, with 
suggestions for successful implementation. Educatiorial 
/ Techno logy. . 1975. r5, 41-44. 

Gaff, S.S. Institutional change. In S.-^. Gaff (Ed.), 
Resource Notebo ok. Washington: Project on Institutional 
Renewal through the Improvement- o f Teaching/ 1976, 



The author provides an overview of institutional 
change along ..with an annotated bibliography of recent 
writings in the area. • " * 

Sikes, W.W., Sohlesigner, L.E., Seashore, C.N. Re^newing 
higher educat ion from within. San Francisco: 
Jossey-Bass, 19'7T7 38-^2. 

This chapter from what the authors describe as "a 
manual for change agents" explores some of the factors 

V 

which inhibit change on campus. 
Resour??e materials that describe teaching and course 
improvement techniques may assist you in determining what types 
of changes are feasible for yoif to make. In Resource File 1C you 
will find; 

The educational development group: IDSP. Learn ing and 
Development . 1976. 7* No. 4. ^ ^ 

This newsletter describes a service availab\le at 

^ \ 

McGill which ^nables attention 4^ be focussed on 
in-class teaching performance through videotaping^ and 
questionnaires completed by processor and ' students . 
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^ Gaff, J.G. New approaches to improve t-eaching. In w. Ferris 
(Ed.), Learner^Centered R eform . San Francisco- 

Jo ssey-Ba sFrT575T~9^5'^^^9T7 

Goldschmid, B. , & Goldschmid , M.L. Modular Instruction: 
principles and applications in higher education. 
^^^rnin g and Deve lopment , 1972, 3, No. 8. 

Goldschmid, B. , & Gcldschmid, M.L. Individualizing 
instruction at. th^ college and university ' level 
• . Learning and Development , 1973, ±, No . 7 . 

^ Roid, G.H. Research on university teaching: a perspective. 
' * ' • Learning and Development , 1970, 2, No. 1 . 

If you are working with a group, or consulting colleagues, 

the answer to this question will likely be obtained through 

discussion. Some guidelines for productive grou^ discussion are: 

1. Circulate'a copy of the questions to be discussed and 
answered. Provide "model" answers as a guideline if 

• possible. 

2. Encourage faculty members to write answers to the 
questions 'in advance. Some of these may oven he 
circulated prior to discussion. 

S. During discussion, use time "limits as a means of 
encouraging "to the point" discussic. , Be prepared to 
be flexible, however, if your time limits do not appear 
to be realistic. 

Have a discussion leader or chairman who moves toward 
specific answers by asking one more question of the 
"what" cr "how" type Ce.g, "What do we want students to 
know?," "How can we change course XYZ?"). 



\ 
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5. Examples are , particularly useful in focussing 
discussion. The disc-ussion leader can ask for specific 
examples when the conversation ir, tending toward 
general it ies . 

6. Have a discussion leader who periodically summarizes the, 
discussion. If the group disagrees with the summary, 
further discussion could clarify the issue; if the group 
agrees, discussion can proceed. 

Resource File ID contains the following relevant article: 

Brilhart, J.K. Effective group discussio n (2nd ed . ) . 
Dubuque, Iowa: Wm . C. Brown Co.,'l^t^, 9^-149. 

These two chapters provide a detailed guide to the 
classroom; the general principles, however, are 
adaptable to any situation. 
There are a number of roles that the consulant can play in 
the completion of the first step. (1) He or she can act as 
discussion leader, assisting in the directing or focussing of 
your answers to the key questions. (?) You can ask a consultant 
for feedback on the answers you haye--are they specific enough? 
(3) If you are unabl^ to judge the feasibility of the size of the 
evaluation you plan to do, or the practicality of the changes you 
would like to make, a consultant can provide an estimate of the 
time and resources required for your plan. 



Question ^: What will be evaluated? 

There are at least three possible "targets" for 
instructional evaluation: teaching performance, course content 



28 



1 1 



and crgariization, and program content and organization. Teaching 
performance consists of the skills that you use while in the 
classrcom--lecturing ability. ability to ask and answer 
questions, to lead discussions, etc. It does not Include the 
content cf^ho course, the sequence in whioh topics are covered, 
the assignment?; that students are given, or reidings. The 
distinction b*n.we(-n the professor as an instructor and the 
content of the course will not be complete, especially when the 
instructor is solely responsible for designing the course. You 
may wish to evaluate each separ,ately, or you may not vrishc'to make • 
the da^tincticn at all. 



Evaluati on "Targ e t s " 

- teaching performance 

- ^-^ourse content and organization 

- program content and organization 



\ 



If you are examining a course or program, briefly list the 
goals of that course or program ,(e.g., to give students an 
overview of American history, to train radio-therapists). 

Ycu m--,y wish to consult one of the following articles 

dealing with defining your course goals or objectives (File IE). 

Cohen, A. Objectives for college courses. Beverley HiUs 
Calif.: Glencce Press, 1970, qZ^If^'f^^^j ^ 

Why write and use behavioral objectives'' 
Learning and Development , 197?, ^4(1). 

•Hess, K.H. The role of objectives and the teaching of 

Voi^^lV^'o^n .5^^^ Com£Osmon and Communication,. 
'Vo, 27^-213. 
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Pascal, C.E. To'wards faeaningful .educat ional ^ object i ves . 
Learn Ing and Developmen t , 1969, .^H) . 

s 

Is the Task Comp lete? 
At the completion of Step One' you should have written 
answers to each of the four questions dealing with the purpose of 
the evaluation. 

1. What is prompting the evaluation? 

2. Will the evaluation be primarily for improvement or for 
personnel decisions? 

3. What . types of' changes are you considering or what 
decisions do you need to make? 

4. What will be evaluated? 

Each answer should be as specific as possible and should be 
the consensus when more than one individual is participating in 
the evaluation^ 
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( STEP TWO 
WHAT ASPFCTS OF TNSTRllCTrON WILL BFl EVALUATKD? 

t 

The Task 

You have now clarified the purpose of your evaluation, 
including a general statement of what you intend to evaluate. It 
is necessary to become more specific in your planning, and to 
define , the particular aspects of instruction that you are 
interested in. 

If you have said, for. example, that . you are evaluating 
teaching performance, you can specify whether you are interested 
in the teacher a-s a lecturer, discussion "leader, etc. 
Essentially, you are * est abl ishing priorities (e.g., it is 
important in the department that professors be effective seminar 
leaders). This further specification of what you are going to 
evaluate will enable you to do a more direct and efficient 
evaluation (so as not to be asking questions about things that 
are not important) . • • 

In the following sections are checklists which are intended 
as guidelines for the specification of the aspects of instruction % 
that interest you. They are neither' exhaustive nor authorative 
lists; feel free to add to or modify them. At the conclusion of 
Step Two,, you should have completed one or more of the checklists 
or generated your own list. 
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If you are primarily interested in the evaluation of 
teaching performance, continue on this pai^e. 

If you are primarily Interested in the evaluation of a 
course, turn to page 20. 

If you are evaluating both teaching performance and course 
content, continue on this page. 

If you are evaluating a program, turn to page 26. 



Evaluation of Teach in g Performance 
How to Complete the Task - 
Teaching include? a variety of abilities or 3k^'lls--the 
university instructor is often, for example, a lecturer, a 
discussion leader, and an evaluator of student learning. In 
order to assess your effectiveness as an instructor, it is 
necessary to examine each of the skills that are relevant to your 
teaching situation. If your goal is the improvement of your 
teaching, you will want to find those particular skills which 

urn, 

need improvement and concentrate on them; if you are making a 
personnel decision, you will want to uncover both strengths and 
weaknesses in your teaching and plan , changes for the weak areas. 

Research. on university teaching h&s attempted to identify 
the underlying components of" effective teaching. Most 
investigations have been based on/ student ratings: factor 
analyses look for groupings in quest iimnaire items (items which 
are consistently rated in the same ha^y are grouped together). 
This research has led to the ident\ f icat ion of four major 
factors ; 
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1. ^^l^^f the ability tc put material across 
interesting, clear and stimulating way. 




2. 



Rapport,. the ability to establish and maintain empathy 
with, concern for, and interaction with the students." 

♦ 

3. Structure, the tendency to have and follow a definite 
course outline or schedule. aeimite 

students^^'' tendency to demand a great deal fro 

In addition, it may be necessary to further separate teaching 
ability into either more specific skills or roles. To ihis end, 
two forms are given on the following pages. The first list 
contains various roles that an instructor may pl^ay. The second 
list was developed as a part of a diagnostic and trairjing program 
at t\L University of Massachusetts (Clinic to Impro^ University 
Teaching). it emphasizes classroom behaviol, and is 
improvyement-oriented . 

If you are interested in adding to the lists, or creating 
your own list, you may wish to consult the following articles 
which discuss the components of teaching. FiU 2A contain. 

^^"''^rin/*^;' ? Penney, M. Instructional Analys is Kit. 
Centre for Learning and dewelopm^tr}^.^GiTTlhn^gr sity , 

Donald and Penney outline twenty-two elements of 
the instructional process, under the headings of: (1) 
?ourse planning, (?) Course Content, (^) Instructional 
Procedures, (4) Learning Materials, and (5) Evaluation 
cf Learning. For each elements, articles and books 
containing further iaformation are given. 
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Eble, K.E. The recognition and evaluation of" te 3chinR . 
Washington, D.C.: Amer icjin , Association oT Unfversity 
Professor?, Project to Improve College TeaehinF'., IO70, 
98-99. . .1 

Eble presents a checklist of ^fective teaching 

ch.iracter ist ics as perceived by students and faculty. 

Hi 1 deb rand ,\M., Wilson,^ R.C.^ ^ Dienst, E.P. Evaluating 
' - university teaching- Berkeley^ Calif-: Center for 
Research and Development in Higher Education , 1 97 1 . 

I.ssacson., R.L., McKeachie , W . J . , Milholland, J*E*, Lin, Y.G., 
He feller , M • , Baerwald t , J . E. , ^ Zinn , K .L • Dimensions 
of student evaluations of teaching. Journal of 
Educational Psychology , 1964, 44, ^ii4-:?5i . 

Using student rating results, six factors of 

student evaluations of'teaching were identified: skill, 

overload, structure, feedback, Rroup interaction, and 

student- teacher rapport. 
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^ Check the items which describe the teaching roles relevant 

t 

to your situation. Add any items 1?hat are not included. 



Lecturer 

Discussion leader 

Adviser for student research or projects. 

Adviser for independent reading 

Demonstrator (e.g., laboratory courses) 

Evaluator of student learning 

Course manager (e.g., modular or CAI courses) 

Student counsellor 

Materials developer 

Team teacher 

Tutorial leader 

Mo*del 



Other roles? 



18 

Form #2: Teaching Skills 




— Establishing a learning set (preparing students to 

learn) 

Logical organization (arranging course content)' 
Pacing 

Elaboration ( clar^4j,f ying or developing an idea or topic) 

— E^iprcssion (e.g., speaking ability) 

Asking questions 

Responding to questions 

A Facilitating student participation 

Closure ("integrating points in a lesson) 

Evaluation of student learning 

Selecting the appropr,iate level of challenge 

' Using a variety of teaching methods and materials 

. Creativity 

Management 

Flexibility and individualization 

_ Interpersonal relations 

' Creating a learning environment 

Stimulating enthusiasm . * 

^ Establishing a frame of reference (perspective) 

, Identifying and clarifying values 

Other skills.? 
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If the roles or functions vary from one course to another, 
or among Individual faculty merabers complete a separate list for 
each variation. 

Discuss the lists with all involved faculty; you do not need 
consensus, however, as separate lists may be used for 
individuals. 

f 

Is the Task Complete? 
When you have completed Step Two for the evaluation of 
teaching, you should have: 



1. One or more completed checklists that speckify 
the teaching roles or the teaching skills 
relevant for your course(s) or program. 



2. One or more lists you have written that 
speclrfy unique teaching roles or skills for 
your course(s) or program. 



You have answered the question from Step One (What is being 
evaluated?) in more detail by specifying the particular aspects 
cf teaching you are interested in. This specification will lead 
you to the selection cf sources of information and techniques for 
collecting information that are most appropriate for your 
situation. 

Please turn to page 31 for Step Three, or continue to the 
next page if you are also evaluating the course. 
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Course Evaluation 
How to Coraplete the 'Task 

Course evaluation has traditionally Included teaching skill, 
without an attempt to separate the instructor's performance from 
the content or organization of the course. Dependent on the 
purpose of your evaluation you may wish to examine a course 
Independently of a professor's skill (for example, in 'order to 
decide whether a course should be deleted from a program), or you 
may want to evaluate both teaching performance and^ course 
content. In the latter case, it is suggested that you also 
complete the previous section of Step Two. 

In designing a course evaluation, there are three questions 
to be answered at this stage. (1) What type of co-urse are you 
evaluating? (2) What are the goals of the course? (3) Wha-t 
aspects of the course do you want to examine? Answers to each of 
these questions will assist in determining the type of 
Info'^raation you need to collect. 

Question 1: What type of course are you evaluating? 

The form on the following page includes common ^ypes of 
courses at the university level. 
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Form #3; Course Type s 

• 

Check the item or iteas which best describe your course. 

Survey course (introductory level) 

Survey course (advanced level) 

— Seminar 

Reading course (independent study) 

Tutorial 

- - Laboratory 

Praoticum or internship 

Lecture/discussion (non-survey) 

Clinical course 

Other? Description: ' 



Other characteristics which are relevant to the type of .course 
are listed below. 

Number of students enrolled in the course 

Number of sections of the course 

Number of professors teaching the course 

e —————— 

Number of teaching assistants involved 

Level (undergraduate, graduate) and year 

Required or elective course ■ 
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pue3tlQn2: What are the goals of the course? 

In order to assesg the ef f eo t i venf>ss of your course, it is 

necessary to^ have ^ clear statement of what the course is 

intended to accomplish. If you have courso objectives or a 

cietailed course outft.ne, they will provide -the required 

information. If neither is available, you may'wish to consult 

one or more of the following books which discuss how to formulate 

course goals^ or objectives. File 2B contains: • 

Cohen, A. Gb jeotives for college courses . Bevef¥*|f. Hills, 
Calif.: Glencoe Press, 1970, 9-64, 73-87. (See File 
IE.) 

Deterline, R.H., &Lenn, P.D. Coordinated instructional 
systems . Palo, Alto, Calif.: Sound Education, \nc . , 
1972, 11-17.^ • 

Kibler, R.J., Cegala, D.J., Barker, L.L., & Miles, D.T- 

Obj ect ives for ins truction and evaluation. Boston: 
AUyn & Bacon, ^1974, , ~ 

Mager, R.F. Pr^eEjl^lL'li IJl^tr jc Uor^ objectiv es. Palo Alto, 
California: Fearson, IQ^. 

When timf^ is limited or you feel that objectives are not 

suitable to your situation, you can u^e the questions on the 

following pages to describe what your'' course is intended to 
accompl i sh . 
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Form //M: Course Goals 



Is the course a "basic skills" course? Yes No 



Examples: performing calculations, problem soWing;, 
drawing techniques, grammar of a second 
1 anguage 



If Sv-) , list the basic skills the student will attnin: 



Does the course provide a general knowledge of an area or 
areas? Yes No 



Rxnmplf^r;: abnormni ps yr* ho 1 f?^ y ^ Furop^an 1 i t ^ t 
. ' i n format ion retrieval sy.^^temn 



If so, list the area( s) 



Does the coursp emphasize an in-depth knowledge of a topic or 
topics? Yes No 



Li St the topic( s) : 



21 



Is it theoretical knowledge? 

Is it applied kncwledge? _ 

Does it include knowledge of research? 



IrlL-'''"' r.T'^ e-PhasUe rese.rc,. skills In an area cr 



If sc , wh.it arensC s) : 



Dees it Involve students doing their own research? 
Does it involve criticizing research in the area? 
Does it include a knowledge of current research? 



crkncwledl:?'^^ IT/^'^' practical, application of a body 



Fx...npi..s: student teaching, psychological testing 
-social work, geological field work 



What is the application? 



PS 



Question 3: What as pe cts of the course do you want to_ ^m i n e? 

Depending on the purpose of the evaluation, you may want to 
emphasize certain aspects of the course, e.g., for a 
required-optional decision, you may be most interested in whether 
or not the course content is an integral p^^t erf the program. 
The components of the course that you choose to emphasize will 
determine the type of information that you collect. 

For m #5; Course Compone nts 



Content (topics included, comprehensiveness, etc.") 

Organization (structure or sequence of topics) 

Textbooks, readings 

Assignments, projects 

Evaluation procedures 

Audio-visual aids (films, tapes, TV) 

Outside resources (guest speakers, clinical facilities 
'Schools) * 



Internal resources (labs, computer facilities 
classroom) ' 

Special methods (modules, computer^assi sted instruction 
student-led sessions, field trips) * 

Other : 
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Is the Task Complf' te? 
When you h;ive completed Step Two for the evaluation of a 
course, you should have: 

1. A completed form! indicating the type of course you are 
evaluating. j 

2. Either a list o,,f course objectives, a detailed course 
outline or a completed form indicating the course's 
intended outcomes. 

3. A list of the components of the courne that you will be 
emphasi zing/' in the evaluation. 

Program Evaluation 
How to Complete the Task 
In order to evaluate the effectiveness of an academic 
program, it is necessary to specify what the program is intended 
to accomplish, and to consider what aspects of the program will 
be examined. The two questions to be answered at this stage are: 
()) What are the program goals? and (?) What aspects of the 
program will be evaluated? 

Question 1: What are the program goals? 

If you have developed a list of program goals it will 
provide the answer to this question. You may want to check your 
list with the list of questions on page 28 to ensure that you 
have all the relevant information. " 

If you do not have a statement of program goals, you may wish 
to consult one of the following articles or books which could be 
used as a guideline in specifying your goals. Resource File 2C 
contains: 
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Graaha, T. Principls and models for assessing faculty 
performance: A monograph. Faculty Research Center. 
Univ. of Cincinnati, Ohio, 1972. 

This is an excellent paper relevant to all aspects 
^ of the evaluation process. The "Goal method approach to 
assessment" described on pages 15-19 is most relevant at 
this point. 

Mager, R.F. Goal analysis , Belmont, Calif.: Lear Seigler, 
Inc./Fearon Publishers, 1972, 136 pp. 

The author guides you through t^e exercise of 
turning vague general goals into specific sl-.tements 
about the outcomes a course or program is attempting to 
achieve. 

Psscal, C.E., & Roid, G.H. A method for generating and 
evaluating course or departmental objectives. Paper 
presented at the annual meeting of the American 
Educational Research Association, New Orleans. Feb. /Mar. 
1973. 

Pascal and Roid present an excellent guide for 
formulating program goals. 

e 

In your discussion of program goals, the questions given below 
can be used as a guideline. Again, discussion should include all 
faculty in the program; discussion techniques given in Step One 
will be useful at this point. 
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Program Goals; Questions to Cons ider 

f. What general knowledge or skills would you expect n student 
who completed the program to have? • 

Examples: be able to solve problems in advanced 
calculus, have a knowledge of modern Russian 
novels, be able to manage the accounting 
system in a business, organization. 

2. What types of careers do graduating students tend to 
enter into? 

3. Is the program emphasis theoretical or practical, or 
some combination of theory and practice? 

i 

Question 2: What aspects of tf^e program will be evaluated? 

Depending on the purpose of the evalution, you will probably 
want to emphasize different aspects of the program. For example, 
if you are making decisions concerning the addition and deletion 
of courses, you will want to examine the role of the existing 
courses in the program, and teaching performance will not be 
r el evant . 

Several components of a typical program are" listed below. 
Consider which of these are related to your purpose; add items if 
necessary. 
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Form g6: Components of a Program 

Program Content 

— . Content of individual courses 

i Structure and format of individual courses 

Relationship among courses in the program 

Relevance of courses to program goals 

Program Methodology 

— Teaching performance of faculty 

Methods used (seminars, labs, independent study, etc.) 



Program Context 

■ Admi#>istrative structure 

Physical facilities 

— Budget (sources, administrative procedures) 



Personnel 

Use of teaching assistants 

_ Staff roles, teaching load 



Other; 
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If your evaluation will include the content or' structure of 
individual courses, turn back to page 20 and complete the section 
on course evaluation. You will need to consider each course 
being evaluated separately. Also, if you intend to examine the 
teaching performance of faculty members, turn back to page 14. 
Some teaching' skills or roles may be perceived as relevant for 
all professors in the program. There will be var iation however , 
and lists should be prepared individually. Extra copies of both 
forms are available from the Centre for Teacning and Learning 
Services . 



Is the Task Complete? 
When you have completed Step Two for program evaluation, you 
should have: 

1. A list of program goals. 

?.. A list of the components of the program that you wish to 
ev al ua te . 

3. If your evaluation includes individual courses and/cr 
teaching performance in the program, you should also 
have completed the appropriate sections. ^ 
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STEP THREE 
WHAT INFORMATION WILL BE COLLECTED? 



The Task 



In Steps One and Two you specified and described the purpose 
and the target of the evaluation. Step Three is concerned with 
the type of information to be collected.' 

There are thYee parts to this step: (1) deciding the sources 
of inforojation (e-.g., students, colleagues, course materials); 
(2) deciding what instruments and techniques will be used (e.g., 
interviews, questionnaires, observations) , and (3) selecting or 
developing the necessary instruments. This is both a complex and 
somewhat arbitrary step. Many possibilities do exist, and there' 
is no one answer Vor all situations. 



c How to C omplete the Task 

— -: — 

Part One: Sourc es of Informa tion 

The selection of appropriate sources of information will* 
depend on the purpose of the evaluation, what you are^valuating, 
the discipline you are in, and the time and resources available 
for the eval^uation. The chart on page 3^ suggests some possible ' 
sources of informatio'n . 

Research on instructional evali>ation has compared various 
sources of information, afid related some sources to other 
criteria of effective instruction. Several summary articles are 
available in Resource File 3A . ♦ 
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Aleanjoni, L.M., ^ Yiraer, M. An investigation of the 
relationship between colleague ratings, student ratings, 
resea>*ch productivity, and academic rank ' in rating 
instructional effectiveness. Journal of Educa tional 
PsyGhology . 1973, 6a, 27^-277. ' ; 

Colleague and student ratings were gathered for a 

group of 477 instructors and then ' compared to the 

instructors' research productivity and academic rank. 

Colleague and student ratings were not significantly 

related to research"" productivity. However, colleague 

ratings were significantly related to academic rank. 

Bergquist, W.H. , & Phillips, S.R. A handbook for faculty 
de velopment . Washington, D.C.: The CouncTI" for the 
AJvanceraent of Small Colleges, 1975, 45-49. 

Blackburn, R.T. , & Clark, M.J. Assessment of faculty 
performance: Some correlations between administrator, 
colleague, student, and self-ratings. Sociology Of 
Education, 1975, 48, 242-256. ^ — 

This paper addresses the uncertainties surrounding 

tho evaluation of faculty work performance and reviews 

the conflicting studies of the two principal 

professional roles, teaching and research. 

Centra, J. A. College teaching : .Who should evaluate it? 
Findings . 1974, j_(1). 

Centra provides a readable summary of research 
concerning colleagues and . student ratings of 
instruction . 

Centra, J. A. The relationship between student and alumni 
ratings of teachers. Education and Psycho logy 
Measurement . 1974, 34, 321-3?ri 

Student and alumni ratings for teachers were found 

to correlate .75. Agreement suggests that student 

ratings may indeed reflect overall long-term 
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satisfaction with instruction. 

fCulik, J. A., & McKeachie, W.J. The evaluation of tenehers in 
higher education. In F.N. Kerlinger (Ed.) Review of 
research in education » (Vol . 3). Itasca, Trffnoisf 
Peacock , 19T5. 

The ^uthors give a recent and comprehensive review 
of research on student, peer, self, and administrator 
ratings of teaching. 

Miller, R.X. Assessing teacher effectiveness. Proceed ings 
£l Intern atioral Conference on Improving "Universitv 
T^acKing . May, 19757 Heidelberg, ?i?-6l , TT^It, " 

Sagen, H.B. Student, faculty, and department chairmen 
ratings: Who agrees with whom? Research in Hiaher 
Education . 197^^, 265-272. ~ — 

The study found little agreement among faculty, 

student, and department chairmen regarding overall 

effectiveness of instruction. * 
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A Guide to Selecting Sour ce s of Info rm ation* 



Purpose 


Target 


Di sc ipl ine 


F 

Sources 


Ifnprovement 


Course 




Co 1 1 eagues 

Course materials 
Support services 
(library, etc.) 






Applied or 
Professional 
Co urse 


Professional 

associations 
Alumni 




Teaching 


: 


Students 

Colleagues 

Alumni 




Program 




Col leagues 
Students 
Administrators 
Ma ter 1 al s 
Graduate students 
(TA's) 






Applied or 
Professional 
Tr ain ing 
Program 


Pro fessional 

associations 
Government agencies 
Community groups 




^ ■ 




Outside evaluators 
Alumni 

Drop-out students 


Personnel 
Decisions 






Colleagues 
Students 
Administrators 
course materials 






Appl ied 
discipline 


Pre fessional 

associations 
Community groups 




* 




Outside evaluator 
Alumni 



*The type of information which each source may'provide is 
summar i zed on page 37. 
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You may also wish to consult some of the more spe ific 

articles in File 3B. 

Centra, J. A. Strategies for i mprovi ng oo liege teaching , 
\ ERIC Clearinghouse on Higher ftfucation, Report 

Washington, D.C.: American Association for Higher 
Education, 1972. 

Centra, , J. A. The student as godfather? The impact of 

student ratings on academia. Educationa l Re searcher , 
2(10) , U-8. 

Centra, J. A. Student ratings of instruction and their' 

relationship to student learning. American Educationa l 
Research Journal , 1977, iH^''^' 17-24. 

Gage, N.L. Student ratings of college teaching: their 
justification and proper use. In D.W. Allen et al. 
(Eds.) Reform , renewal , reward . Amherst, Mass.: 
University of Mass., 1975. 

Menges, R.J. The new reporters: students rate instruction. 
/ Teaching News and Notes (Univ. of Kentucky, Office of 

Instructional ResourcesT, 1977, 3(3), 3-8. 

y 

Murray, H.G. Limitations of students ratings of college 
teaching. CAUT Bulletin, April 1977, 28. 

Although the selection of appropriate sources of iTi format ion 

is complex, some general guidelines can be given. These 

guidelines are based on both practical experience and research 

results. 

1. Students should be included as a source of information, when 
possible, regarding both their learning and their attitude 
toward the in t struct ion . 

2. It is advisable to use more than one source of information 
as a "cross-check." Information obtained from' one source 
(e.g., students) may be influenced by a number of 
circumstances (e,g,, the content of one particular lecture 
or exam). Use of more than one source will allow you to 
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a. 



* 

check for the influence of unusual Circumstances. for 
personnel decisions, it is Pssenti..l that more than one 
source be included. 

3. Course evaluations should • include course materials 
(outlines, exams, assignments), as well ^ students as 
sources of information. 

The source of information may determine the outcome of the 
evaluation. Biases, both positive and negative, can be 
built into the evaluation through the selection' of^ the 
sources. It is suggested th|t you read the , relevant 
information from the resource files and have your consultant 
view the proposed sources to assist- you in 'identifying 
potential biases. 

Different sources are able to provide, different types of 
information; for example, you cannot expect students to be 
content experts, and colleagues are not likely to be able tos 
judge your rapport with students. The table on page 37 
summarizes some possible types of in formation 'that you can 
expect -from various sources. 



\ 



Sc urce 



Po s i b 1 e T n f o a t i o n 



Co 1 1 eagues 
St udent s 



Admin i strators 



A 1 um n i 



Go V er nmen t 
<=igeno ies 

Commun i ty 
agenc ies 

Facul ty f rom 
ether un iversil ies 

Univer'sity servi^.es 

. , library, 
ocmpt.;t^>r cen^.r p) 



Course content, textbooks, 
resources 

Ratings of effectiveness of 
various teaching skills 

Difficulty cf course 

Textbook, exams, assignments 

Clarity of course objectives 

Achievement 

Relevance cf course or prcgr-^'n 
goals to the profession 

Extent to which graduated 
students .ire prepared 
for profession 

Extent to which course or 

program prepared them for career 

Curriculum requirements 
Professional requirements 

Employment trends, type cf 
training required 

bourse Of* pre gram ccntent 
or organization 

Frequency cf student us-^, 

preferred titlf=>s ^ programs, <^tc. 
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For m &7 : So urces of Inform atio n 

^ Check the sources that you will include in the evaluation. 
Adil ncurccs if necess.iry. 

— \ Chairman 

- ^ Self 

____ t ud e n t s 

Administrators 

— Pre fessicn al ev aluator s 

A 1 urn n i 

Gcvernment agencies 

Community agencies 

. Research/ to ac hi ng assistants 

Co ur 3e/ prcj.? r am materials 

Drcp-cut .students 

Faculty from other universities 

_ University support services (library, drcp^in centres, 

computer centre) 

Other; 
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Fart Two : Techniq ue s for Col T e c t ing Info rmation 

Several techniques Per collecting evaluation information are 
described belcw. ^ 

^- Ill^^f>Li^_ll? face-to-f?iCA, individual question nn<i nnswer 

sessions. They may be preplanned and fairly structured or 
ccrapletely open-ended and flexible. They" are most 
appropriate for obtaining information that cannot be easily 
quanti/ied, or foP uncovering unexpected feelings and 
reactions. 

2. Obser-y :jt ions involve watching the instruction in progress. 
Specific behaviors may be observeo and recorded. The 
emphasis is often on the occurrence of the behavior; however, 
ratings or judgements may be made. Observers must be trained 
and seme systematic procedure used for record ing"* ' the 
observations. 

^' Commen ts may he formal cr informal records. Individuals or 
groups can be asked to record their feelings, reactions and 
attitudes to various aspects of the course, program or 
r.eachinc^. Comments mav be collected during discussions, 
meeti-igs or classes. This technique is useful for detecting 
siO'-' effects or unexpected reactions. 

^' ^-^y include ratings of instruction o^^ 

cpen-enied que-sticns. When large numbers of people are 
invcivei, questionnaires are r qui^k, practical, and 
tren^rallv reliable means of collecting certain kinds of 
i n fc r-nnt ion . Hcwev/er, q ue s t i onn ;t i r es do, as all other 



techniques, have limitations. Tt is always advisable to use 
them in conjunction with another method. Questionnaires give 
you ratings'" of instruction: they provide information on 
attitudes toward, or judgements about the teaching or r^ourse. 
Extensive research has been done on tlie use of 
questionnaires: variables that influence ratings, 
relationships with student learning, the ways in which items 
<^an be grcuped, and reliability over time. j 

Reviews of this research and a few "classic" articles 
can be found in Resource File 3C. 

Centra, J.A^ Student ratings of instruction and their 
relationship to student learning, American E ducational 
Research Journrq , 1977, Jj4, 17-24. 

This study randomly assigned students to sections 

cf a colleg^ course, and investigated the relationship 

between teacher-produced achievement and student 

ratings. A moderately positive relationship was found. 

«"^stin, F., Greenough, W., ^ Menges, R. Student ratings of 
^u-^llege teachers: Reliability, validity, and usefulness. 
Review of Educational Research, 1971, 511^535. 

Feldman, K.A. Consistency and variability among college 
students in rating their teachers and courses: A review 
and analysis. Rese^arch in Higher Educa tion , 1977, f> , 

Fe Idman , K.A. Co use characteristics and college students* 
ratings of their teacher s---what we know and what we 
don't. Research in Higher Educa tion, I97ft, 9, 19Q-?4^. 
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Kulik, J. A,, i McKeachie, W.J. The evaluation of teachers in 
higher education. In F. Kerlinger (Ed.), Review of 
Research in Educ ation , 1975, l(^) y 210-240. 

Che ck l ist s are lists of behaviors to be observed, or choices 
to be made. They are efficient for recording occurrences of 
characteristics or Jsehaviors, but do not usually provide an 
opportunity for ratings or judgements. 
^- l££k^ student learning are usually a part of the course, 
and may be used as one indication of course effectiveness. 
However, many variables influence student learning; it oannot 
be used as the sole criterion of teaching effectiveness. 
' Conten t and task analysis are techniques for analyzing a 
course outline, exam or lecture notes. They will give you 
information about the content or structure of a course. Task 
analysis emphasizes the type of 'learning that is expected of 
the student and the appropriate sequencing of instruction. 
^' -Xilg^QAapga and ta pe recor dings provide a lasting record of 
observations. No additional information is obtained, but the 
tapes can be reviewed and carefully analyzed. 
When is a particular technique most appropriate? Your 
choice will depend on a number of variables: the type of 
information required (for different purposes); time and resources 
.ivailable; ?ind the sources of information. General guidelines 
for selecting the technique are summarized in the chart on pages 
and 45. Resources from which this information was summarized 
and elaboration on some of the techniques are contained in 
Resource File 3D. 
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Cronbaeh, L.J. Evaluation for course improvement. Teacher's 
College Reoor d , 1963, 6^4, (Pages 8-21 deal"~wrth 
methods . ) • 

Donald, J.G. The evaluation of learning. Learn in g and 
Development > 1976, 8, No. ^ 

Roid, G.F!. Towards a system of course evaluation. L earning 
and Development, 1971, 2, (2). ~' 

Roid, G.H. Learning about ratings vs. rating learning: some 
questions about questionnaires in course evaluation. 
Learning and Deve lppment , 1971, 3, No. U. 

Smock, R., & Crooks, T. Diagnosis and remediation of 
instructional problems witho-ut the use of standardized 
instruments. Paper presented at the Annual Meeting of 
the American Educational Research Association, New 
Orleans, 1973. 

Some additional ideas can be found in pp. 12-15 of Tony 
Grasha's "Principles and models for assessing faculty 
performance." Copies of this article are in Resource Files 1C 
and 2D. ' 

For each source of information (colleagues, students, etc.) 
you have selected, record the technique (interview^ observation, 
etc:) to be used in collecting that information. 



Part Thr ee: Selec ting or Developing Instruments \^ 

•The Instruments used in the evaluation det(>rmine, tc a large 

\ 

w 

extent, tne validity of the results. Great care must be taken to 
ensure clear, unambiguous and objective questions. Choice of 
wording cr response' categories can . influence results either 
positively or negatively. 
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^3 

instruments. Sovf^rnlof thesf'nre ^w.-^ilable in fh^ vPsourc« 
files. W^u>n possible, it i.s r occmmended th t such instruments, 
ur questions from them, be used. "Tf items must be writt-n for a 
unique situation, have your consultant review them. 
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' ^"i£LZi_lg.iig.tin^_Techni^ for Col le ctiriR In formation 



Techn ique s 


WMer: to Use? 


For which 
Source? 


Time 
Required 


Resources 
Needed 


Examples of Use 


Ill V t? r V 1 c ws 


to obtain de-- 
tailed indepth 
information on 
a small number 
of topics--to 
answer ^^why" 
rather than 
"what" 


f ac ul t y 

students 

admin i s- 
tr ator s 


1 arg^e time 
investment 
f or 

gather ing 
and 

analysis 


inverv iewer ( s) 
• 


course evaluation 
(analysis of course, 
relevance to student 
career plans) 


Observations 


to obtain de- 
tailed infor- 
mation on spe- 
cific behav- > 
iors/skills/ 
' per formance 


f acul t y 


1 arge time 
investment 
for 

gathering 


trained 

observers 
• 


teaching evaluation 
(to deterrriine whether 
questions are used 

aDDroDri^^t^"! v^ 


Comments 


to obtain un- 
expected 
side-effect 
information 


all 

sources 


moder ate 
time invest- 
ment for 
analysis 


none 


evaluation of an 
innovation , e.g., 
mod ul es 


wuest ion-' 
naires 

6S 


to gather 
informat ion 
on attitudes, 
reactions , 
perceptions , 
from large 
numbers of 
peopl e 


all 

sources 




moderate 
time for 
preparation 


pr int i ng 

ready made items 
(optional ) 

computer 
anal ysi s 
( opt ional ) 


teaching evaluation 
(to determine stu- 
dents' perceptions 
of in-class teaching 
performance) ^ 

o3 
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Summar y : Selec tin g Tech nique s for Coll ecting I" ^^^n^_gt ion ( continued ) 



Technique 


When to Use^ 


For which 
Source? 


Time 
Required 


Resources 
Needed 


Examples of Use 


Checkl ists 


to determine 

of charac- 
ter i St ics , 
components , 
behav iors 


all 

ou ur c e o 


short time 
c onim X um en u 


none 


course evaluation 
(to examine the com-* 
pr ehensi veness of 
a course, outline) 


Ac h i e V etn e n t 
tests 


student 
1 earning 




ill Uvu c I c; U vT 


struction 
computer 
anal ysis 
( opt ional ) 


course evaiuabion 
(to see if course 
objectives are 
being mastered) 


Task 

anal ysi s 


cour se 
mater ial s 


1 Cl V U i. \^ J 

subject- 
matter 
experts 


Ui UU C I a L c: 

time 

investment 


pruceuures 
for 

compl et ing 
an analysis 


course e v al u^t ion 
(an analys'fs of a 
lab mani^^i developed 
for the' course) 

r 


Videotapes 

and 

tape 

record ings 


detailed 
information 
on specific 
behaviors/ 
skills/per- 
f orinanc.e 
(where a 
lasting re- 
cord is needed 
for later 
anal ysis) 


f acul ty 


large time 
investment 


eq^uipmant 

trained 
observer s 

equi pmen t 
operators 


teaching evaluation 
(to analyze ability 
to facilitate 
d iscussion) 
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Good instruments are important .in any evaluation. Extra care 

should be taken (e.g., the use of more than one instrument, 

having colleagues review the instruments) when personnel 
decisions .iro being made. 

Turn to the sections dealing with the techniques you '^rf^ 
u i n I.; : 



Teohn i que Page 
'Achievement Tests (student learning) 46 



Checklists nj 

Comments/ Rev iews * »43 

Content or Task Anal ys is i49 

In ter V i^ws i^q 

Observations ^1 

Que ionn.i ires t:^^ 

V-ideotopes c;c^ 



Achievement Tests (measures of student learning) 

Measures of student learning are usually a part of the 
course. In ^n evaluation, you will most likely be concerned with 
overall results of n course, although it is possible (for 
teaching or course improvement) that you would want to examine 
student achivement on a specific aspect of the course. 

The f T ; "1 wi His resources give pr actio a"" advice, on the 
development of achievement tests. Although th^^y ^^-p p,-,^ 
contained in the Resource Files, they are available fr your 
' • n 3 u 1 t a n t . 

6(; 
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Cranton, P. A. Construe tins tests. Centre, for Learning and 
Development, 1977. See Teaching and Learning Modules, 
N 0 . 6 . 

This module goes through the step-s involved in 
''hoosing, constructing nnd evaluating different types of 
tests, 

Donald, J. Objective tests. Office of Educational 
Development, Mc<Tril University, 1Q77. 

"^is pamphlet gives guidelines for the construction 
of multiple choice, true- false, matching ^^nd short 
answer tests. 

Green, J. A. Teacher Made Tests (?nd Ed.). New York: Harper 
i Row, igiT. 

Green discusses different types of tests and gives 
guidelines on how to construct each. The book is 
intended for public school teachers but can easily be 
applied to the university level. 

Chjecl<l_i^t_r. 

:'>iru-- checklists are usually specific to the situation, it 
will likely be necessary to construct your own. So|ne guidelines 
are given below and some examples are available in Resource File 
?E. f 

1. Items should be short, cle?)r and un amb ig -jo us . 

^. Items should contain only one behavior or example f 
tf>achinr-^ performance. 

Items should be objective, an.\werable in p. "y^'-.-no", 
"ex ist-doesn ' t exist" mr-nner. 
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Wi^h long or complex checklists, it is us^-^ful to 
organize, the items into catef^orier, or o sequonnp t--. 
f .''c i 1 i t 9 te use . 
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Comments and Review s 

Using comments or reviews for gathering ciata involves 
collecting extended verbal or written responses to either 
specific or general questions. Gathering such data is relatively 
easy, but interpreting the rosults is a challenging task. 

Several approaches are possible: 

1. Ask for specific, written, anonymous reviews of a 
particular aspect of a course or the teaching 
performance. 

P. Ask structured, unbiased questions on a written form 
(similar to a "written interview"). 

'"-The more structure that is imposed on comments or reviews 
ur>ed in an evaluation, the easier the analysis will be However, 
useful inf^rmati can be obtained by accepting or soliciting 
completely open comments. 

Some general guidelines are: 

1. He flex ible--accept ail information that is relevant to 
your evaluation. 

rp.^p.-^nsive and open to unsolicited oommenta. 

If you use a structured format, be clear, specific and 
obi?^-tive in your question^. 

s 

Resource File 3F' contains information on how to collect 
•v-mmonts an i reviews and some guidelines on how to interpret this 
1 n f r m a t i O' n : 

B.^gdan, R., ^ Tnylor, S. fntroduo tion to qualitative 
research methods. New York: John" Wiley ^ Sons foTs" 

Th*- n\;thors p>-ovide R.JideMnes for the u.^e and 
a n a i y .s i s --^ f c omm e n t r- . 
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Shefn.in, T.M., & TayU^f, S. A foniKit. ive iippronoh t stixiont 
ev.iluDtion f i n^tr uo t i.^n . K<iuc;_ational TocIui-tIo^v, 

This article 1 1 1 ustr .it.es the u^p r>f comment ^ in nn 
ev c^l ua t ivHi . 



Cotitent and Task Analysis 

Content analysis is the systematic description of the 
content of a cor.mun ic at ion (textt^ook, course outline, lectures, 
films, etc.). Categories and sub-cateRor ies are .agreed upon, gll 
topics ;-ire arrangf^d into these categories, and the relationships 
among categories are examined (e.g., do some necessarily precede 
others in order to be understood, is there a sequential or 
historical order, is there o pr ac t i c al- theor et i c al continuum?). 

Further information is given in Resource File '^G : 

Sax, G. Em p i r i c a 1 foundatio ns o_f e ducationa l research. 
Knglewood Cliffs, N.J.: Prentice Hall, 196 8," ?7WTq"."~ 

Task analysis emphasizes the skills required of students, 

uii vhe r^-i ;,t lonshi pr. among ttiem . A task analysis specifies, in 

order, tne steps the learner will be taking wh^n he is 

sucopssfully performin<7, the task. Beginning with the final goal 

)r objeotive of a segment of instruction, you ask "What skills or 

knowleri^e are required of the student before h^ can a^hi^vf- thi^ 

goal?" Repeat this question for each skill or knowledge 

statement until you have reached the low^-st level which is 

included in the instruction. The list c?.n be arranged In a 

h i er arch i.-:^ al diagram. Examples and m-^re detailed instructions 

are given in Resource File 3G : 
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Anderson, K.C.. K Kanst , G.W. Fd uo n t i .■> n 1 p.syc ho 1 or y . On.i.f 
Me,^.i, S7_8^. " 

Hnvis, R.H.. Alexander, L.T., Yolon, S.L. Lea rning system 
d_esi_3n. New York: McGraw Hill, 1Q7U, 1 30-1 . 

Gagne, R.M. Learning h i er ;5rchies , Fduoationnl Psychologist 
T^68. 6, 1-9. ^ -b-— ' 

^^ax , ii. Empir ic a l foundations o f educational research. 
Eni^lewood Cliffs, N.J.: Prentice Hal 1 , 1968 , 273-?7q . ' 



I n t er V 1 ew_>: 

Interviews vary considerably, dependinc; on the information 

bpxv.^l sou^^ht and the person being interviewed. Tt is advisable 

to use a "structured" or a " sem i- st r uc tur '^d " interview format, 

that Is, to plan the questions, or at least the areas that will 

bf^ covered. I f ^ more than one person is interviewing, it is 

important thit the content of the interviews be consistent. 

'".orno ,\rn,-r:\] ^ 1 1 1 d p 1 i n f> s for planning interviews are: 

1. ne spcrnfic and direct--ask what you want to know, 
leaving no r-i,nrn for interpretation. 

f^e clear-- avoid "jargon" or vague phrases. 

^, . Bf^ cone i se--peopI e are donating time. 

. Ba flex ible--useful information can be obtained if 
people have an opportunity to express themselves freely 
at some po int . 

S. Be ob.1 PC t i ve--dcn ' t "lead the witness." 

Provide training or practice for the interviewers. 

possible, record interviews, or at least provide a 
cl'-->ar form f:r categorizing responses. 

• -f possible, have the interviews conducted bv a 
" neutr a I " pe^- son . 
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■ R^-souroes in File V\ cont;.in further information on 
inter V lewi n?^ : 



MoGray, E. Plnnning and conducting 
-lIl^.^L^l^ws . Austin, Texas: Lear n i-n^~Concept s ,~Q7?r: ^ 

Morgan. H , \ Cogger, J. The interviewer's manual . The 
Psychological Cor poratioPTTl 97X 

Sa\,G. Obtaining in format ion from respondents- T-the 
lnt^:^rvlew and the questionnaire. Empirical foundations 
-Ji educational research . Englil^^d Cli'ffs' " — W~J~ 
Prentice Hall, 1968, 201-213. . ' 



. Ob ser vat ions 

It is essential that observations be structured: the 
observer .-nust know which b^^haviors to look for and have an 
^^bjective and consistent means of recording, the observations. 
Teohniques whinh ■ are used to record observations are described 
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Cneoklists may be uspd (see p. ^^7) whore specific 
behaviors are listed and checked off as they occur, 
i.itegories of performance may be defined clearly, and 
".bservers may check the appropriate category at timed 
interval s. dur ing the observation. 

Observers may actually evaluate or judge various aspects 
of a oourse or teaching style based on pre-defined 
criteria and using a rating scale;.' 
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K Wh.it ovfM* rnot fi vi .It* 'l^tpr* V :it 1 -ui (Mi.^^u^n, it i 

impor* t in t t..^ ^^fn^^k t.h^'* i^Mis i ^U.^^nc^ v '"^ f -'^b 5;^r v 1 1 i n s . 
H.we t w.^^, obsf^r V f^r s wntoh th^^ snme behavior f^-^r n sh.-^.rt 
period of timp .ind check the rigreement of their' 
obser V at ions . 

. If possible^ have an '^unbiased^^ or independent person 
obser ve--^espec i n] 1 y if teaching skills are being rate<>. / 
^- ''St^ minimum number of c ategor ies--- 1 1 is difficult to 
remember^ and usp a large number of categories. 
. Trnin y^^iif ':)b nor v rr s- - 1 et them ' '^practice" before 
j^^luilly usin^, the results of their observations. 
^> • '^^-'^ .simple 3 record Inp, form ns possible, espec^inlly 
^ if ^^bser vat ions nre timed--the observer should not be 
(^^n.-ontr at inR on finding a place on a form,, but r.a^her 
r)n the teachinp, performance. 
K'^s^urce Fil'^ contains tv;o relevant items: 

Instr liyr. ional Developmen-t Service 'Project. Devel opmen t of a 
C^Atee>or y System . McGill University, 1975^^ 

An observation system developed at McGill 

University is d^scrib^^d. 

Mjrtin, I. Thp development an(i us a of classroom observation 
inst.r uments. Cana dian Journ al of Education, 1 977, P, 

\ 

The author discusses the development and use of 
: b v a t i :) n systems. 
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Ques tion n a Ire s 

Student questionnaires have been used and researched 
extensively; ccnsequentl y , many forms are available. 

Resource File 3J contains several questionnaires which can 

be used as they are, or adapted to suit a particular situation.. 

Berg.jtiist W., ^ Phillip., S.R. A handbook for facultv 
deveiqpjnent. Washington, D . C .T TFTe ^ To-unc i f" foV'T^-e" 
Advancement of Small Colleges, 1975, 5?-86. 

Illinois Course Evaluation Questionnaire & Manual. 

'"'''a'coLan:'' I r^^^N^^^/^^^ (self-report for. to 
accompany S.I.R. form), Educational Testing Service. 

Wood P. The description and evaluation of a colleffP 
department's faculty rating system. Paper presented at 
the annual meeting of the American EducatLnal Research 
Association, New Yo rlL^v Apr il 1977.' "^j^earcn 

""7^ ^instr^uclion^^J^ ^"Tmpro^re-^t ''''^-^l^ 
?:;t?n11:-tr"e:\^n%^atL^!"°^^ Un . v er si t y , \o un ^e^Jf^g^^^ 

Copies of questionnaires from the fcUowing institutions are 
1 1 '^c .^v n ] ablp in Fi ] e M : 

i^''"^'^ ^^^'^^'f'i'^S <-^nd Learning Services McGi U 

University ^ - ^ , nt. u 1 [ i 

Dawson College, Montreal 

Michigan State University 

'-'niversity of Wisconsin ^t Green Bay 

J^) iy«rsity of T 11 i no is 
'Hiversity of Iowa 
'diversity cf Arizona 
Pr inneton l)n 1 versi ty 
Mniversity of Washing ton 
Virginia Ccnmonwea] th University 
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If ycu are wr i t ing your own quest ionn aire items, several 
resources are available. 

1. Centre fcr Teaching and Learning Services, Questionnaire 
Service, 1979. A consultant will assist you in using a 
computer-printed questionnaire based on the particular 
aspects of instruction that you have selected in Step 
Two . 

2. Resource File 3K contains exerpts from books and 

articles which provide practical guidelines and 

examples. These resources do not deal exclusively with 

student questionnaires and' will be most useful if you 

are constructing questionnaires intended for other 

sources of information (colleagues, professional 

association, etc.). 

- Henerson, M., Morris, L. , & Fi t z-gibbon , C. How to meas ure 
attitud es . Beverley Hills: Sage, 1978. 

McCallon, E., & McCray, E. ' Designing and using 
q j est io nnaires . Austin, Texas: Learning Concepts, T9757 

Sanders, J., & Cunningham, D. Formative evaluation: 
selecting techniques and procedures. In S. Borich, 
(Ed.), Evalua ting e ducational programs and products . 
Educational Technology Publ ication ,1 974 . 

Sax, G. 7- Empir ical fou n dati ons of educational I'esearch. 
En^^lewood Cliffs, NTJ.: Prentice Hall, 1968, 2l5'-237\ 

Zimbardc, P., k Ebbeson, E. In flu enc ing attitud es and 
changin g behavior . Reading, Mass.: Add i son-We si e"y , 
1968, T?3-T?^ 

* 

V 
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V Id eo tap e s / 

Since videotaping is a technique for recording obs^^rvat ions , 
the information given in the section on observations (page 51) is 
relevant here. 

For detailed information on the use vjf videotapes, contact 
'the Teaching Improvement Specialists at 392-8320. 



Is the Task Complete? 
At the completion of Step Three, you will have: 

1. * Selected the sources of information that you will use. 

2. Selected the techniques for collecting information. 

3. Selected or developed any instruments or forms which 
will be used. 
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STEP FOUR 

WHAT CRITERIA WILL BE USED FOR DECISION OR CHANGE? 



fhe Task 

Before the data are collected for the evaluation, it is 
important to consider how they will be used to make decisions or 
to plan changes. This step is analogous to formulating research 
hypothese3-~if criteria are set before the information is in 
front of you, it will lead to more objective and systematic 
decisions. Admittedly, setting criteria for the evaluation of 
instruction is a complex and somewhat arbitrary task— many 
variables affect student learning, student ratings, and all other 
indicators of effective instruc tio.n . It is possible, however, to 
set standards, or a range of standards, keeping in mind that the 
measurement will not-be precise, and that you may need to make 
revisions to these standards as the evaluation continues. 

You may find, as you attempt to set criteria, that this 
process acts as a "review" of your evaluation intrumen ts-.-i f you 
cannot determine what type of responses should be obtained from a 
question, perhaps the question requires revision, or does not " 
need to be asked. 

The process of setting criteria varies depending on the 
purpose of the ev al uat ion-- for example, making an "error" is much 
more serious if you are making personnel decisions than if you 
are evaluating your own teaching in order to make improvements. 
Three evaluation purposes will be consid^-red separately. 
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- If you nre evaluating teaching performance and/or a course 
for imp rovement . continue on this page. 

- If you are evaluating a proi^ram , turn to page 62, 

- If you are making a personne l deoisio n , turn to page 65. 

Teach ing and/or Course Improvement 
How to Complete the Ta sk 
When the decision being made is whether or not to make an 
improvement, the consequences of a "wrong" decision are not 
serious. It is probably preferable to set criteria low: at 
worst, this will lead you to make Improvements that are not 
actually necessary. Available time and resources are the major 
considerations. A general procedure is given below. 

1. Usinf^ Lhe aspects of instruction (teaching skills, 
course components) that you specified ;n Step Two along 
with the items or questions that you selected to measur^^ 
thc^ aspects, categorize or order them according to 
importonce. For example, if you are teaching an applied 
course designed for professional training, field 
4 experience may be the most im-ortant course component; 

if you are teaching s small discussion-oriented class, 
your skill as a discussion leader may have first 
pr icr ity . 

.?. Ccnsidering the time and resources ^/vailable, d^^cidp th*^ 
r^aximum number cf a'-eas in whioh changes could be made. 
Some changes are obviously mere time consuming than 
others (^.g., revising course mater i.ils ver jus speaking 
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so 

more clearly): 'you may wr-int to c riteR or* i 2f» the (^h.-inRef; 
(e.g., two l.ir^e change.'' nre feasible, or fivo sm.^ll er 
oh iriRes) . 

f 

Consider which sources of information are most ^ 
impor tan t-- i f you are considering course oontf^nt 
changes, colleagues' opinions may be most vali'i; 
however, for a particular teaching skill, students may 
provide the most relevant information. 

Predict the responses that you will ^et and/or rate 
your sel f . 

Criteria may now be determined in a number of ways: 

a. Specify percentages or frequencies of responses 
below which you will change, e.g., if at least ^0% 
of the students are satisfied with the sequence of 
topics in the course, no change will be made. This 
win need to be done separately for each area being 
evaluated and all sources of information need to be 
considered. Criteria will vary according to the 
priority of the area. 

b. If it is not possible to set specific crit.eria for- 
some or all of the aspef^ts of instruction, you may 
consider using, a "range" for a ciiterion: for 
example, you could say, "If between '^0?. and SO^ of 
the questionnaire responses are negative, students 
will be interviewed to obtain further information," 

t 
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Change 



Fur ther 
Ev aTuat Ion' 



No Change 



Number of Positive Responses- 

I — ■ ^ 

or, ycu could say, *'If the first five interviews are 
all negative, 1*11 stop interviewing and plan to 
make charig*»s; if .the first -five, interviews are all 
positive, T^SLJ make no changes; if two are negative 
and three positive, 1*11 continue Interviewing/* 



Number of 
Negative 
if esponsef^- 




Numberof Positive Responses 



Specify the arenas in which you will work, e.f^., 
regardless cf the level 'of responses,- changes will 
be made in the lowest three areas only* 
f>1: screpanc ies between expected ratings and obtained 
ratings may ^Iso be used » A large discrepancy could 
1 '^n<i to further evaluation, or ecu'' d be used to 
ohange . pr ior i ties (e.g., if you thought you 
presented course objectives clearly, but students 
did lot, making a change in th^t area may become 
mere important) . 
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A_Nct(>_on tht' Po tent i.i l Dange r s_c f_0 ve r - 0 uj^ 1 iT i c a t i o n 

All cf this may seem highly quantitative and you may object 
that the complex and often loosely-defined things with which you 
are de^Tllng do not lend themselves to numbers. Indeed, 
quantification may seem to be a violation of the nature of the 
things ycu are trying to examine. We are sympathetic to that 
view and do not want to encourage trivializing the valuable 
things in education in order to come up with a set of numbers. 
What we are suggesting, though, is that the attempt to set 
criteria is in itself a valuable step to go through and it is art 
exercise which provides a picture of what is actually considered 
valuable. urthermore, it will prevent evaluation from becoming 
simply a' bureaucratic exercise. It is when you think of the 
consequeoces of the evaluation that it moves from being a game 
into being a respected part of the improvement process. 

Is the Task Comp le t e ? 
I^i9.^LQIL-£!ll^o fl_Cou£se_ IjT^ o v emen t 
When Step Four is completed you will know when improvements 
will be made, based on each of the information-gathering 
. techniques you have chosen. For each aspect of instruction being 
evaluated you will have: 

1. 1 numbf^r cf responses, a score, a frequency, etc.- which 

will be the standard for making improvements; or 
P. 1 number of areas in which change- .;iu be made 
regardless of "absolute" results; or 
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. 3. a range of numbers of responses or scores which will 
lend to a decision to make changes or a decision to 
continue f»v;ilu3 1 ion. 

Program Ev alu.itlon 
How to Comp l ete the Task 
Setting criteria in a program evaluation is complicated by 
the number of sources of information available-, the variables 
influencing decisions (cost, university regulations, etc.) and 
the number of inter-related decisions to be made. A general 
procedure is described below. 

1. Referring back to Steps One and Two, list the decisions 
that are being considered. 

2. Referring to Step Three, what information will be used 
to make each decision (e.g., which items on each 
questionnaire, which questions from the interview, 
etc. ) ? 

^. WhMt other considerations will affect each decision? 
Examples: 

- time and resources required for the change; 

- university regulations or requirements; 

- requirements of a professional association, 
certification board, etc.; 

- personnel available (teaching assistants, 
f acuity workload) ; 

- physical facilities. 
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What are the ••costs" of making a wrong decision 
(deciding to nake a chan^^e when it isn't required, or 
deciding " to make nc; change when it should bp made)? 
Which error would be the easier, one to accept? IT the 
most acceptable error is to change when no change is 
necessary, set the criteria lower. If the most 
acceptable error is to make no change when it is really 
needed, set the criteria higher. 

Criteria may be set in one of the following ways: 

a. Percentages or frequencies or positive responses on 

an item, group of items, or question, abov^^ or below 

which no change will be made. 

Cr i ter ion 



Chfinge 


No Change 


i ; 





Number of Positive Responses 



b. A criterion could be a range of numbers , further 
evaluation being required if the obtained results 
fall into that range. 
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Fu r ^ h V 






Fv .'^ 1 i: ' t i rn 





Numlv^r c f Po s it i V e Rf* spc n s^"' s 



N iim bf^r c f 
Ne^ativo 
Re sno n so s 




Number c f Po si i". i R^*^ spon sp^ 



Th''» '^■^'V n-i I 1 1 ii'^t r nt-. j '-an hr- t imp- s^i v i nr^. 

V a ] u 1 i c n c c n n 1 n ij^ s . 
F c ^ i n fc'-rmnt i c n wFii^^^ 1^ nc^ qu^nt ri'?bl' , yctj cnn 
s^^lr^-t typ.^s of infcrm.Ttirn thnt wonld ^-^opt^iblo 
: nc^ no-^Dt. ^b I f- ; yen mav wish -^iblisH ^ 

■minimum I^vpI of qiirnlity Mir^^. ^'-<^ptab^^. 

nr.nn^,r:,^r y ^ Yc u mny wi's^ :nvclvf^ -/^cn ^.iP * ;vi ^ in 

YCMjr d i so iJ S S 1 G n S , rr r^^^n^i^a •^cmTlAnts ^rcTi n 
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Is the Ta s k Comple te? 
Progra m Evalua t i c n 
When St.^-p Four is completed you will know how each decision 
will be made, based on each of the information-fathering 
t^ohniqu^.-. you hnve chosen. For each deicision and each 
technique you have either: 

1, A number of responses, a score, t frequency, etc. which 

will be the cut-off point for your decision; 
?. \ range of frequencies or scores which will indicate 
either a decision to change or a decision to continue 
evaluation; o'^ 

3. A non-numerical criterion stated in terms of the type or 
qu.-lity of response or performance that you will accept. 

Per 30 nne l Fva luat ion 
How to Compl ete th^ Task 
Tn p-='''sonnel '^valuation, the decision to be made is usually 
straightfcwTri- the process of setting criteria is complicated by 
the psychological "cost" of making a wrong decision and the 
inflexibiUty of the decision once it is made. 

General guidelines for setting criteria are outlined below. 
1. Determine priorities for the areas of instruction being 
evaluated. Consider sotting criteria higher for the 
mere important areas and lower for the .lesr important 
areas. 



Ccnsiiier whether any var tables wi 1 1 in fluence the 
evalu;ition (e.g., unusu.-^lly large classes; teachiriR in 
an unfamiliar area, or teaching a new course; timing of 
the evaluation, etc.). 

Whenever pCsSsible, incorporate improvement into the 
criteria; i.e., evaluation is r-epeated every semester 
and changes are noted, or evaluation is conducted near 
the beginning and end of a semester • 

Realize that no evaluation gives completely accurate 
measures of the effectiveness of instruction and 
compensate for this as much as possible: incorporate 
more than one measure into each criterion; when a 
discrepancy occurs, evaluate further. 
Criteri^i may be set in the following? ways: 
a. They could consist of percentages or frequencies of 
positive responses for all items or questions that 
relate to a particular aspect of instruction. 
De term i ne the number of areas wh i'?h may fall below 
criterion before a decision is made. Di sc r ep-^nc ies 
amcn.c. sources of information or among instruments 
should l^ad to further evaluation. 
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Measure //I 
Measure //? 



Cr i t.fM' i ori 



c i s i c n 1 




Do i s i 0 n ' 

















Number of Positive RespQn55es 



b. \ criterion could be 3 range of numbers, further 
evaluation being required if the obtained results fall 
into that range. 





Further 




Dec i sicn // 1 


Ev 3l uat ion 


Dec i sicn n? 



Number of Positive Responses 



1 . 



For information which is not qu.inti f i ;^bl e , vo'^ 'T^n 
select types of information that ;gre acoept-^bl- or 
not acceptable, or establish ,t minimum level of 
qunlity thc-t is accept-ible. Yc u may wish to discuss 
this situation with consultant. 

Yc u m.-sy consider relntlvA judgement combined with 
more ibsclute criteria, - . . , if se-eral professors 
are t-aohing similar course? (in terms of content, 
class size, etc.) results obtained may be ccmpar^^d. 
Criteria can be percentiles or other compara^ive 
statist i-s rather than absolute numbers. However, 
-auticn is advised: many variables ar^- Influpncing 
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ev. ligation ^^vsults, ind measuremt^n t?; ■ \'en thoUf?,h 
quant i ri<-^blfs contain a oertciin nmount of error. 

Is the Task Complete ? 
Pers onn el E valuation 
At the .completion of Step Four, criteria should be set for 
your decision, consiiering the priorities of the areas being 
evaluated and the contribution of each source of information. 
Criteria may be in the form of (a) a specific cut-of f . po int (for 
each area with a pr e-determined number cf areas which musf be 
. above that cut-off point, (b) a range of numbers, leading to 
either a decision or further evaluation, (c)' for non-quantifiable 
information, a decision as to what type of' information is 
acceptable, or- (d) relativf^ information from the evaluation of 
more than one instructor. Whenever possible, improvement should 
be considered in tho noAp i f t o ? t ic n cf criteria. 
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STEP FIVE 



CC1.LECTING, ANALYZING, AND 
INTERPRETING THE INFORMATION 

The Task 

The evaluation plan is now complete and ready to be 
l-nplemented . In Step Five, the information is gathered and 
s^ummarizpd . Interpretation of the results fellows from . the 
criteria established in Step Four. 



How to Complete the Task 

Part One: Coll-ect inR the Information 

^_ ^ . 

Data collection is a relatively str ai(?ht forward procedure, 
however, the result's c^.i be influenced to some extent. Research 
has shown that responses can be affected >v S-'uch variables as the 
intructions given during administration' of a questionnaire, 
anonymity of answers, timing (immediately before or after an 
examination)., etc. The- following points illustrate some of the 
problems that can arise when gathering information, and 
procedures that should help ensure accurate and valid collection 
(and hence interpretation) of information for the evaluation. 

1. Many factors can bias the inf'ormant. Where possible, 
therefore, questionnaires, comment forms, etc., should 
be completed anonymously. In some sitOations, 
'^•specially for course improvement, it may be useful to 
have signed evaluations in order to follow up on 
specific comments, or to relate ratings to ability, 
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achievement, etc. 



Interviews, observations, 



etc 



should be conducted by someone who is objective and who 
will not be directly affected by the results of the 
ev al uat ion . 

Less obvious biases may result from some oversight like 
an inadequate provision of time for the thoughtful 
completion of forms or questionnaires. Similarly, the 
sample of respondents may be affected by the procedures 
used to gather information. For example, it is 
generally inadvisable to have individuals take forms 
heme, or to mail them back--the returns will be biased 
s?^mpl e . 

The information gatherer (e.g., an interviewer) can plsd 



influence the results. The inexperienced interviewer, 
for instance, either may not elicit genuine student 
reaction to what war. actually taught or may sw;]y the 
answer by accidentally suggesting an expected response. 
Be sure, therefore, that any students or assistants used 
to make ratings, observations, etc., are adequately 
trained. Whenever possible, check the extent to vvhich 
such assistants agree, or are consistent in their 
j udgptnents . 

Scorers or raters may also (often uncono iousl y ) reflect 



biases in the way they grade tests or read questionnaire 
responses. Using more than one scorer and ohec.-ing for 
consistency helps to reduce tne problem. Tt can be 
eliminated by the use of machine scoreabl*^ answer cardr. 
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for questionnaires whenever suoh n • proceiiur^* is 
/■ 

practical . 

Part Two: Analyzing the Informa tion 

Methods for aralyzing information vary for each instrument 
or technique used. Brief comments will be given for sf»veral 
types of instruments, and some resources will be listed. It is 
suggested that the consultant participate in this stage of the 
evaluation if any difficulties are encountered. 

> Achievem ent T ests 
The analysis of achievement test results depends on the type 
of test being used. An essay test, for example, might be 
evaluated by comparison to a model answer; ^ multiple choice test 
simply involves totalling the number of correct t esponse . 

Guidelines for scoring the various types of achievement 
tests can be found in Resource File 5A. 

Cr3n:on, P. A. Constru cti ng testes. Centre for Learning and 
Development, McGill University, 1977. See Teac hing and 
Learning Modu les , No . 6 . ' 

Green, J . A . T eacher-m ade tests. New York: He^rper & Row. 
1 975, 

Check! ists 

The analysis of checklist results simply involves 
determining the frequency of checks for each item cr category of 
items. There may be both positive and negative items on the 
checkl i sts-- i f sc , total these items separately. 
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Comm en t .s/ Re v i e W5 

Depending; .-^n tho t.yp^^ c-nmmf^nt.s ^"^r reviews col lf^r>t p/l ^ th^^ 

m.^ lysis m,iy invv^lvo: 

K Ca t e^VM' i zi nr. the comments nocor-^i i ru^, to the i r^suo or 
t*.>pic, r^.s wpH as the direction (positive or nep;ativp) 
of the oomment. 

■?. Putting, the comments on n scale; averaging or tot^llinp, 
ratings. 

V. Record ing onanges in the d iro^-tion or n;itur e of 
comments, if they are collecteQ over a period of time. 

I n t r V 1 *^ wn 

If tfie Mitf^rview is structfjred or sem i-- r^tr t ur ed , r o spo n r-^ 
will be rf^.-.^rdfvl \\^v each quest,,] on, ^ot,er;ory, or topi--^. Th^ 
rospvu^s^^n ( u\ be c a tpf^.or i zed or , rated: they may be "positive," 
**neg.aive/* .ir ^'neutral;*' they mav be "yes/^ ^'no" or "nn!;^ee," 
''d:r.mree'' M^per. v) f responses. Responses cnn be listed nni wh^n 
tti'^y ire r , the number of oecurrencer. of enoh r ecor '1 . 

M^re detailed in form it ion con be found m Resource Filn t*>B. 

Mc Co lion, VlA .y ^ McCray^\ TlA^^i^I^ condor- ting 

in Ler V ie ws , Austin, Tex.js: Lear^nin^' concepts , '^1 OTs^; 

^•>rn,.jr^, M., '■^%P/.^,f^r , J. The in L^r v [ewer manuoi, Th^^"^ 
P^' v^ho 1 op, i r ol Cor por at iohV 197V- ' 

"^^^>^', * Kmpirh'al L'? [1^1.^1 'I ^.i: ?l ^n'K:ationai research. 
Fn^l^>w<Vo"d ^CVi ffo, : PFent i e^H cTTf "TQ^^ / ^PO T-^" JT;^ 

Ob se r V a i:^'!^!^ 

Oependinp; on the n^-^-iir^ of y^ur ob s^^r v at. i. .)n s , = n-.- vio-y-i- 
m '^y ^'••)n s\ st .-^ f : 



1. Recording the frequency of occcrrrence . of various 
•observations. 

2, Averaging rcitings of judgf-ments mnde for each it<^m. 
Quest lonnair es 

Generally, questionnaire analysis consists of recording the 
frequencies of responses for each item. Items which are 
measuring the same teaching skill or course oomp'onent may be 
averaged; percentiles may be calculated in order to compare 

« 

instructors. 

Information available in Resource '3C inclu. .?s: 

Cameron, A>M. Mult\le cjioice :^l^Ml£tion and course 
H.i'^tion, user's Ruide. McGill Co~rnput ing ' re"n tr e'~ 
August 1977. ' ~~ fs - t.. 

This guide explains (with examples) McGill' s system 
for .-veering course evaluation questions. 

Kul i k , '-v^. A. Student reactions to instruction. Memo to the 
!l-?.^,MLt_y i^^A' University of Michigan, "CelTtr?" TS7 
Research on Learning and Teaching, October 1976. 

Kulik discusses the use of questionnaire dat-(. 

Content and task analysis are used to determine if all the 
ipp cpriate material is being covered and if it is being 
present-d in the proper sequence. Once the analysis has been 
done, it is necessary to compare this ideal analysis with what 
i"tu;-llv exists and decide where changes should be mad p. 
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Part Three : In t -^r p r et at ion * 

Referrirw; to the criteria s^^t in ?t^ep Fo\ir, determine from 
the results whit decisions or improvements will bei^ade^ A^^ain, 
it is impcrtnnt to remember thnt the dat.i are not^ "per feet'* : many 
vjri.ibles havje^^ influenced the result^. You may find that the 
criteria are unrenlistic: it is acceptable to revise them if this 
L the oaso, but this must not be unilateral. The concurrence,,^ f 
nW involved in setting "the original criteria is required: Tt 
may be necessary to collect further information; your consul anr. 
will be able to discuss the situation with you and \ suggest 
possible a ct^l ens. 

If ycu are using evaluation results for a promotion or 

tenure deMsicn, 3 usf^f:jl r^^cuice (File SP) will be: 

Teaching Ff f ec t i v ene.ss Committee of the CAliT, Guide to^ the 
te_achi^np, dossier: its preparation and use, T)rTft f^te^oFtV 

"^h I s i^iii-ir^ ficntains pr :i:^tica1 su<?, [.^, est. i on s fcr both 
* ri^^ 1 n ^ -M^ pr ^^t..^ t i on of information arid th^ presentation 
r pva]uat ion results in a teaching dossier. 

Is thf^ Ta sk Complete? 
Ar the .'n^ ^npl Ic^n of Step Four you will hav^ collected p^W 
information, inalyzed c:" summarized it in a way that is relevant 
to your criteria, and made decisions based on the crite^i.^.. 
1 n t^r pr eta t, i c a ' and d^p 1 sicn-maki ng will have included discussion 
am cnR all involved iniividuals. 



STKP SIX 
MAKING CHANGK.S 

Th e Tn s k 

Impl «='ment.inp. l.hf> impr <-inon t s in(iicat<-»d by the f^v ;^ 1 un t i v^n is 
1:1 impu-t un nt.op in the prooens: this is the final ^onl of 
oo'uUii'.l i ru', \\\ tn' il unt ion . Un f^)r tun a 1 y , it is also thf st^^e 
where '1 1 f f 1 o ul t i (vs are often encountered. You've d<^ci.i>d to 
improve- y Hir lecturing 3bility--" how dor^s ohp go .^boul 

In ^tt^p Six you will revir-w the time in.i r o 'Svvur ^ . ^- r :iv:i- 1 »> 
t'! aotjolly impIfMnf^nt changer^ and pi .^n , in der-ii wFi-t wlii 
done with tJie evaluation results. S^^condly, materials, -irtioles 
ind r^<-:-.ujror-r. ,ivoil;ibW^ ,it MeGill ore d^--.oribod to ps-.i -.! v^n in 
m.ik i n^^'^i- h.Kir,^';; . 

J w t ^1 Com p 1 e t. e < . h p T .-i :s K' 

■i.'lL_'i^L'^ • i JiiilY. i^^w of Tim e_ a xvS _R ?j^o ur- c. e 

How rni;oh tirr,.e and wh.-Jt --esouroes do you hnvr-" to nnko th^^ 
improvements you nave planned? If you are plqnnin^ n teoohinp, 
improvenonl program, you wU be concerned mainly with y.;ur own 
time oni the avai 1 abl il ity of resources. Tf you are embarkinj; on 
o.^urse revisions you may also be concerned with contributions 
from students, cierioal assistanc?, or the budget for mat»^r(al.s, 
"quipmenr, etc. Pr.)p,r;i!n '-happ-^t^s will likely rpquirf- -a -i^trnied 
analysis of all resources in the department- Tf you nro mokin^ r-i 
promv)tion or tenurr- deci'sion, the fist part of this stop may not 
be relevant. Otherwise, oomplpte pither Form // '^ --^r ^'^ , depend inr 
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depprnlin^^r cn your purpose 



Form //8 

R/;vio w c f Tim e a nd Re sources f or T*^^ 3 c h i n 
^' C o ur s e 1 m pr o v om en t 



t. i:TiO are you wi 1 1 ing to contribute to t e^ohlnt^ 
i fTTprc Vf^mf^n t? ( hour s/days per week/ month) 



How mu^h tim^ will be required (If any) from your student <? 



Ml w iiu^M ^ im^:^ (if nny) wi]l be rpquired Vrvm your oo11e.ip;ues 
^ c t> :;^*r V ) t ] c n of I se 3 , d i !::e u s i 0 n , etc,)? 



■/^^•■n" e i t.f^rrns of bocks or rr.at^^" i .^"^ H?"e nvailablr^ 

^c vi';, or wh-it budget dc you have for books or m^tprlals 



Wh.jt. r r>::i. urf?e'; in t^rms of equipment, audiovisual aids, 
i^'-^rniM^^ aids. ^^tr., ar^^ avaiU^iMA cr what budt^et is 
ivai 1 >b ! 



9.0 



F.i Mil ty members who will 
cor)t»Mbiito timp (if '■}ny 
are required) 



No . of ncur s/da vs 



St. U'i ent r> ( undor^v^aduate) 
w|ic- will ocni.ribute tin^e 
( i f ^" .-^qij i r ed ) 

( 



No . of hour 5/ days 



St ijienLs ( Rr-odiKite) who 
will ccTi tribute timf=* 
(if r equi red ) 



No 



c f hcur s/dny^ 



Re se H r o I-)/ t e.ic h i n^; rf s s x 
t ;u) t ^ whc will oc^n t** i - 
biii.t^ time (if ^^equired) 



Nc . C f be ur s/ d qy r. 



CI ^r ieal nssi sr.-^nt who wi 1 1 



c n ^ r i b u t e t 1 m ^ (if re 



d) 



Me . of he ur .s/ d nys 



9c 
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What is your budget for (?»hanges? 

Time per led ' 

Equipment — — — 
Re search/ Teaching Assi stants 
Clerical/Secretarial Assistants 
Books/I^aterials • 



Dupl icat ion/Pr in.t ing 
Oth^r 



What resources are already available that you may be able to 
use (where relevant)? 



Fquipj:nent 



.Books/Mater ial s/Learning Aids 

Library Resources ^ 

Lobcratory Resources 

Physical Space - 



S 
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P art Two: Resource's' Available for Making ImpVovenient s 

A number of re'sources are available through McGill's Centre 
for Teaching and Learning Services (including the Instructional 
Development Service Project), and the "office of University 
Teaching and Learning. , * « 

,1. The Centre for Teaching and ^LearninR. Servicers has 
collected an ex-tensive^set of books and arjticleS on most' 
• aspeGts of instruction. .^The resourpe. centre (rop^' 

MacDonald-Harrington Buildjng) "is open to all McGiir 
faculty. Your 'consultant will be able to . guide you to 
the appropri-ate mater ials, or you' can c-ontaot the 
Centre's, director at 392-8320. ' . 

2. The Centre ^ 0 ffers an individualized modular course on 
"Teaching and* Learning." Modules may be purchased at 
the bookstore and used independently, or an instructor 
mny register in the course and meet regularly with the 
^course manager to discuss issues arising in the modulef?. 
Topics covered . are: 



- Introduction to the Basic Principled of Systematic 
In St ruction 

- Writing Learning Objectives ' 

- Designing Instructional Modules 

- .^Characteristics cf Effective Instruction 

- Student Motivation; Fostering Positive Attitudes 
Toward Learning 

Constructing Tests 
Grading 

CO'urse Evaluation 

- The Tntruc^or and Student Skills 
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Several gen^^al workshops "and seminars a.re offered by 
the 'Centre each ye^r ; they are. advertised in the 
McGill Reporter Also, any department c^an retjuest Pi 
workshop on a 'specific topic. * • " /. 

McGi4.1's- Instructional Development Service Project 
"provides a 'pro<;edure for improving cl-assroom teaching 
per fornjance.. Instructors meet individually .with a 
teaching improvement .specialist to plan the changes. 
•The process includes the usq of stude'nt feedback on 
teaching performance, an analysis. of course objectives 
or outline, classroom observat iqn , C by the teaching 
•improvement ♦ speciglist) .and videotaping . of 
instruction.' Project personnel may be qont?5cted at 

'McGill' s Office of University Teaching and Learning 
of fers" .financial assi stance^for seme course or program 
changes. Further information and grant ""application 
forms are available by calling- 39^.^^091, , 
Centr'e consulants are' available to' work ^ith 
individuals, groups or departments' in making teaching, 
-e^urse and pirogram improvements. Your evaluation 
,consultant may be 'able to act in this- capacity, or 
refer you to another consul ant.' 



\ 



T. s the Task Compl^ ete? 

< Upon «'cmpletit>n of 'step Six, you 'will hgvr' ( n) a«senspd the 

time and rosoiit^^?. available \'.o you for making ehang(>s, ;ind (b) 

obtained the information and/or 'assistance required for 

f 

implementing these changes. 



r 



CONCLUSION 

/ 




The evaluation of instruction is, a continuous^ process. Tt^ 
•does not '.end when you Kave ^ made one or several improVements . It 
is impt^rtant to constantly "monitor" or evaluate instruction: ' 
Students* needs,, interests and abilities change; co.urse 
materials^ class sizes, and physical facil'ities change.' . ThesJe , 
and ether - variables i/ifluence the effectivenep of your 
instruction. It -wi 1 1^ not , " o f course, be necessary*' to constantly' 
'repeat all steps in the evaluation process. You will likely find 
that, the planning decisions made (Steps One to >our^ remain 
-fairly constant. 



